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Abstract 

Today’s students are growing lip in a world of constant connectivity, instant 
information, and ever-changing technological advancements. The increasingly 
ubiquitous nature of mobile devices among K-12 students has led many to argue 
for and against the inclusion of these devices in K-12 classrooms. Arguments in 
favor cite instant access to information and collaboration with others as positive 
affordances that enable student self-directed learning. 

In this study, 706 middle school students from 18 technology and 
engineering education classes worked in groups of 2-3 to complete an open- 
ended engineering design challenge. Students completed design portfolios and 
constructed prototypes in response to the design challenge. Classes were divided 
with some allowing access to mobile devices during the study and others not 
allowing access. Additionally, randomly assigned classes completed the design 
portfolio electronically, and others completed the portfolio on paper. Final 
student portfolios and products were assessed and assigned a rank order using a 
method of assessment called adaptive comparative judgment. Thirty student 
interviews were conducted as well as 6 teacher interviews. Statistical analyses 
between student access, portfolio type, student self-directed learning, and 
student achievement were conducted. Findings showed that student self-directed 
learning was independent of mobile device access during the study. Mobile 
device access was significantly correlated with higher student scores on the 
design portfolio, but mobile device access was independent of student scores on 
design products. 

Keywords: Mobile devices, self-directed learning, middle school, technology 
and engineering 


Need 

Today’s K-12 students, sometimes called “digital natives” (Prensky, 2001), 
are growing up in a world connected through technology. They are expected to 
be part of a global society that is linked through technology and to possess skills 
that will enable them to excel and continue as life-long learners (Johnson, 
Adams, & Cummins, 2012; Prensky, 2007; Tulagan, 2013; West, 2013). 
Today’s students often have more computing power in their personal mobile 
devices than their parents had during their educational years (Lenhart et ah, 
2015). A recent study from the Pew Research Center (Lenhart et ah, 2015) 


- 2 - 



Journal of Technology Education Vol. 29 No. 1, Fall 2017 


found that “73% of [American] teens have access to smartphones” (p. 5) and 
that “92% of teens report going online daily—including 24% who say they go 
online ‘almost constantly’” (p. 2). 

Leveraging mobile devices to positively impact student achievement and 
self-directed learning, as well as the potential pitfalls associated with mobile 
devices in the classroom, has been a topic of recent discussion (Elder, 2009; 
Johnson, Adams, & Haywood, 2011; Lloyd, 2010; O’Bannon & Thomas, 2015; 
Quillen, 2010; Schenker, 2009; Shuler, 2009). However, the possibility of 
utilizing mobile devices to enhance student self-directed learning has not been 
explored. 

Self-directed learning, a process in which individuals take the initiative to 
diagnose their own learning needs, identify resources for learning, and then 
evaluate their own learning (Knowles, 1975), is becoming increasingly relevant 
in today’s educational landscape (Mitchell, 2014). There is a potential for 
mobile devices to facilitate self-directed learning. As Fahnoe and Mishra (2013) 
noted, opportunities for learners to be self-directed are often experienced with, 
and as a result of, technology. 


Purpose 

The purpose of this study was to determine what effect, if any, the use of 
mobile devices (e.g., iPad or smartphone) had on student self-directed learning 
and achievement in a middle school technology and engineering education 
(TEE) classroom during an open-ended engineering design activity. The 
findings from this research may benefit school administrators, teachers, parents, 
and students as the ongoing debate regarding the inclusion of mobile devices in 
the classroom continues. On a larger scale, the purpose of the study was to 
inform policy and decision makers as the face of education continues to change 
and evolve with the rapid advancements in technology. Mobile devices are one 
example of a potentially educational technology—an addition to the classroom 
that may facilitate learning and improve performance (Januszewski & Molenda, 
2008). The two research questions that guided this study were: 

1. What relationship, if any, exists between middle school student access 
to mobile devices and student self-directed learning? 

2. What relationship, if any, exists between middle school student access 
to mobile devices and student achievement on an open-ended design 
problem? 

Although this study specifically looked at the influence of access to mobile 
devices on student self-directed learning and achievement, it should be noted 
that the findings of this study should not necessarily be confined to mobile 
devices. Mobile devices most directly offer the added benefit to students of 
access to information in real time, communication, and other functionalities. The 
findings from this study can be used to inform current thinking and inquiry 
regarding the place, use, and implementation of mobile devices. On a larger 
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scale, these findings can be used as another resource in the debate surrounding 
personal access to the Internet, communication, and other functionalities in 
public schools. 


Self-Directed Learning 

Self-directed learning has been identified as a key 21st century skill 
required for students to succeed (Fahnoe & Mishra, 2013; Partnership for 21st 
Century Learning, 2015; Zsiga & Webster, 2007). However, the majority of 
current research related to self-directed learning is about adult learners not K-12 
students (Fahnoe & Mishra, 2013; Liu et ah, 2014). Self-directed learning 
combines an understanding of what is not known with an understanding of what 
activities need to be undertaken in order to obtain the needed knowledge (Van 
Deur, 2004). Self-directed learning includes “self-managing, self-monitoring, 
and self-modifying capabilities [, which] . . . characterize^] peak performers in 
all walks of life” (Costa & Kallick, 2004, p. 52). 

Self-directed learning has been identified as positively correlated with 
numerous characteristics, including GPA, openness, conscientiousness, 
emotional stability, extraversion, optimism, career-decidedness, work drive, life 
satisfaction, and self-actualization (Lounsbury, Levy, Park, Gibson, & Smith, 
2009). In one study, Fahnoe and Mishra (2013) examined sixth graders’ self- 
directed learning as it corresponded with technology use. Utilizing the Self- 
Directed Learning with Technology Scale (SDLTS; Teo et ah, 2010), Fahnoe 
and Mishra (2013) reported that students in the technology-rich environment 
were statistically significantly more self-directed in their learning than their 
classmates in the traditional classroom, suggesting that technology carries with 
it the possibility of increasing and encouraging self-directed learning in K-12 
students. 

In their article “Students’ Perceptions of Self-Directed Learning and 
Collaborative Learning With and Without Technology,” Lee, Tsai, Chai, and 
Koht (2014) found "that students who reportedly engaged in SDL [self-directed 
learning] and CL [collaborative learning] in face-to-face contexts also engaged 
in these forms of learning in technology-supported contexts” (p. 425), 
suggesting that self-directed learning practices may occur independently of the 
presence of technology. Exploring the influence of technology on the self- 
directed learning practices of students was one goal of this study. 

Mobile Devices in K-12 Education 

Literature related to mobile devices spans a variety of settings, devices, and 
definitions. This article focuses on mobile devices and mobile learning in K-12 
classrooms and used Traxler’s (2005) definition of mobile learning-, “any 
educational provision where the sole or dominant technology is a handheld or 
palmtop device” (p. 265). Additionally, this study utilized Kim, Holmes, and 
Mims’ (2005) definition for mobile wireless technology (or mobile devices): 
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“technology that provides continuous accessibility to users anytime, anywhere 
without using a wire or cable to connect to networks (like the internet), transmit 
data or communicate with others” (p. 55). For this study, the two identified 
definitions were combined to define mobile learning with the inclusion of 
mobile devices: “any educational provision where the sole or dominant 
technology is a handheld or palmtop device” (Traxler, 2005, p. 265) “that 
provides continuous accessibility to users anytime, anywhere, and without using 
a wire or cable to connect to networks (like the Internet), transmit data, or 
communicate with others” (Kim, Holmes, & Mims, 2005, p 55). 

Despite the rapid increases in mobile devices, mobile learning, and 
educational technology opportunities, research related to mobile devices in K- 
12 settings is limited in scope (Cheung & Hew, 2009; Hwang & Tsai, 2011; Liu 
et ah, 2014; Sutton, 2011; Wan, 2011). Although the impacts of mobile devices 
in K-12 classrooms are relatively unclear (Cheung & Hew, 2009; Hwang & 

Tsai, 2011; Liu et ah, 2014; Sutton, 2011; Wan, 2011), there have been recent 
discernable efforts at implementing more “mobile friendly” policies and 
incorporating mobile devices into student learning experiences (Hwang & Tsai, 
2011; Liu et ah, 2014; Lloyd, 2010; Quillen, 2010; Schenker, 2009; Shuler, 
2009). The benefits of including mobile devices in K-12 classrooms seem to 
center around student access to information, others, and technology (Lenhart et 
ah, 2015; Prensky, 2007; Robledo, 2012; Shuler, 2009; West, 2013). 
Interestingly, Mentzer (2011) showed that access to information (i.e., the 
Internet) did not improve student designs in an open-ended engineering design 
challenge when compared with other students without access. Relatedly, Pieper 
and Mentzer (2013) found that students with access to the Internet during an 
open-ended design challenge spent significantly more time accessing 
information than their peers without Internet access; however, this additional 
time was not always productive or impactful. This study aimed to add additional 
insight to the question of whether or not access to mobile devices, and in turn 
information, will be impactful on student learning in open-ended engineering 
design challenge settings. 

Adaptive Comparative Judgment 

In this study, the adaptive comparative judgment (ACJ) method was used to 
assess student product and portfolio performance. ACJ is a relatively new form 
of assessment, originating in the United Kingdom, and this is the first time that it 
has been used in a middle school research study in the United States. ACJ was 
developed through work by Alastair Pollitt and Richard Kimbell (see Pollitt, 
2004, 2007, 2012 and Kimbell, 2007, 2012) and relies on comparisons rather 
than rubrics or scores for assessing student work. ACJ, based on Thurstone’s 
law of comparative judgment (Thurstone, 1927), is a form of assessment in 
which judges are presented with two different artifacts of student work (in the 
case of this research, the judges viewed two design portfolios or two student 
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products). Each judge is not asked to grade either of the artifacts but rather to 
simply make a holistic judgment about which artifact is better based on a 
provided rubric and their own professional opinion. This process is repeated a 
number of times until a rank-order is produced for the artifacts viewed by the 
judges. 

Arguing in favor of this form of assessment, Pollitt (2004) and Kimbell 
(2012) point out that although current trends in education often favor rubrics, 
assessment of any kind ultimately involves the comparison of one thing to 
another. 

All judgements are relative. When we try to judge a performance against 

grade descriptors we are imagining or remembering other performances and 

comparing the new performance to them. (Pollitt, 2004, p. 6) 

Following the theoretical development of the ACJ process, a grading engine 
was commercialized by TAG Assessment under the name CompareAssess. 

Using a complex algorithm, which has been validated repeatedly and used on 
thousands of student artifacts (Pollitt, 2004, 2012), CompareAssess combines 
rankings from a panel of judges to assign a final rank order to each artifact. In 
the CompareAssess engine, each artifact is compared with other artifacts by 
randomly assigned graders until a specified reliability requirement has been met. 
The reliability obtained is best understood as the judge consistency coefficient— 
similar to an inter-rater reliability level (Pollitt, 2015)—and this method of 
assessment has repeatedly demonstrated more reliability and validity than 
traditional methods of assessing student work (Kimbell 2012; Pollitt, 2004, 

2007, 2012). 

An additional point mentioned in the ACJ literature relates to the method’s 
validity; ACJ results were compared with ranking results through traditional 
methods, and the resulting value of R 2 was 0.81, corresponding to a correlation 
of 0.90 (Kimbell, Wheeler, Miller, & Pollitt, 2007). These findings further 
suggest that the ACJ method of scoring is valid and will produce highly 
correlated results to traditional marking. 

Interestingly, although ACJ is not widely used in the United States, the ACJ 
method of assessment shares some similarities to other innovative assessment 
techniques being piloted. Denson, Buelin, Lammi, and D’Amico (2015) recently 
published their work on developing a creativity assessment that makes use of an 
online platform for viewing and rating pieces of student work. Although this 
method did not use ACJ, it had other functions similar to CompareAssess and 
demonstrates a larger interest in alternative and more effective methods of 
assessment. 
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Methodology: Mixed-Method, Quasi-Experimental Study 
Pilot Study 

Following Internal Review Board approval from the participating schools 
and the university, a pilot study was undertaken at a local middle school with 
two classes (the Exploring Technology class for seventh and eighth grade 
students). Each class received the same instruction and completed identical 
design challenges. One class completed the design portfolio on paper without 
access to mobile devices, and the other had access to mobile devices and 
completed the same design portfolio using an iPad app for portfolio creation 
entitled LiveAssess. LiveAssess was developed concurrent to the ACJ assessment 
engine through the efforts of Kimbell (2007) and similarly commercialized by 
TAG Assessments. The purpose of the pilot was to experiment with the research 
process, instruments, teacher pacing guide, and design challenge. During the 
pilot study, the researcher took copious notes regarding minor tweaks, language 
changes, and areas of confusion for the participating students. These notes, in 
addition to student comments regarding possible improvements (as gathered 
through post-pilot student questionnaires), were all used to revise the study prior 
to full implementation. 

Research Design 

Implementation of the full study took place in a large suburban school 
district in the western United States. This large school district is one of the 50 
largest school districts in the United States with a primarily suburban middle- 
class population (16% free or reduced-price lunch). A total of six teachers were 
recruited for the study based on willingness to participate and possessing similar 
characteristics (teacher license level, similar years of teaching, similar classes 
taught, similar school facilities, and recommendation from the district TEE 
coordinator). Each teacher agreed to implement the study in at least two sections 
of the Exploring Technology class, an introductory TEE course for seventh and 
eighth graders. A total of 706 students were included in the study, which 
required five class periods (90-minute class periods every other day for 2 
weeks). A total of 18 classes of the Exploring Technology course from the six 
teachers formed the population of the study. Two teachers used paper portfolios 
with their classes, and four teachers used iPads to complete the portfolios via the 
LiveAssess app. Four randomly assigned teachers (i.e., one paper-based 
portfolio, three iPad-based portfolio) were instructed to allow ubiquitous mobile 
device access, and the other two teachers (one paper-based portfolio, one iPad- 
based portfolio) were instructed to prohibit this access during the unit. The 
counter-balanced nature of the access and portfolio medium was undertaken in 
an effort to highlight possible problematic variables related to the dependent 
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Teachers were trained prior to the study in a 2-hour training session 
conducted by the researcher during which teachers were provided with paper 
and electronic access to all study and training materials. Teacher compliance and 
fidelity to study measures and to the provided teacher script were monitored 
through daily observations by the researcher and by means of responses to 
qualitative interviews at the conclusion of the study. 

Students began the study by completing a pre-study questionnaire. This 
questionnaire included demographic questions, inquiries regarding their use and 
comfort with technology from the Digital Natives Assessment Scale (DNAS; 
Teo, 2013), and questions adapted from the Self-Directed Learning with 
Technology Scale (SDLTS; Teo et al., 2010). Following the pre-study 
questionnaire, students received instruction related to mobile device use, digital 
citizenship, and the engineering design process. Students were then placed in 
groups of 2-3 to complete an open-ended engineering design challenge. This 
challenge involved the designing of a new container or dispenser for distributing 
pills to patients in specified quantities and at prescribed times (see similar 
examples in Kimbell, 2007, 2012). Students designed the product for a specific 
user: an elderly individual who enjoys traveling internationally. 

Initially, groups of students were provided with a “handling collection'’ 
consisting of materials chosen to stimulate idea generation and creativity (e.g., 
zippers, ties, string, plastic, and clay). Students were also shown pictures of pill 
holders and containers as well as the student creations from the pilot study. 
Following this brainstorming activity, students returned the materials in the 
“handling collection'’ and were provided with new materials from the “modeling 
collection’’ that was used to construct a solution to the design problem. 

Following prompts from their teachers, students filled out a design portfolio 
(either on paper or electronically) throughout the design challenge. The overall 
progression through this activity was managed by means of a provided teacher 
script that instructed teachers when to prompt students to complete a portion of 
their portfolio and when to move to a new portion of the lesson. The design 
portfolio was loosely influenced by similar portfolios used in Kimbell’s research 
(2007, 2012) and was crafted to help the students display their progress through 
the design process. 

Students worked on their designs and portfolios for four class periods and 
on the final day (Day 5), students turned in their portfolios and products and 
completed a post-study questionnaire. Teachers identified five students for the 
researcher to interview: two "high-performing” students, two “low-performing” 
students, and one “average-performing” student. The researcher conducted a 
semi-structured qualitative interview with these students and asked them 
questions related to self-directed learning, mobile devices, engineering design, 
and their experience with the study. Teachers were also interviewed and asked 
similar questions in an effort to further explore, clarify, and highlight the 
findings from the study. 
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Mixed-Method Data Collection 

Following the completion of the study, all the student products were 
collected, and a digital picture was taken of each one, resulting in 175 images of 
student design products. These pictures were uploaded to the CompareAssess 
ACJ engine for later use. Each paper portfolio was also “digitized” using a 
scanner and iPad to record student responses from the paper portfolios into 
electronic versions via the LiveAssess app. These were also added to the 
CompareAssess engine. Data responses to the pre- and post-study questionnaires 
were conditioned and matched, resulting in a pre-study, post-study, and 
combined data set for later analysis. 

Student and teacher responses to qualitative interviews were transcribed and 
analyzed using descriptive and thematic coding following recommendations by 
Saldana (2013). In the first step of this process, the transcriptions were analyzed, 
and several words that described the contents of the response were identified. 
The second step in the process involved a second review of the transcribed 
responses in which the descriptive responses were analyzed for general ideas 
and themes. In the third step, the identified themes and ideas were synthesized 
into overarching themes for each response. These themes were checked for 
triangulation with topics relevant to the study (e.g., mobile devices, self-directed 
learning, and open-ended problems). Following the quantitative analyses, the 
resulting final qualitative themes were used to clarify, expand, and inform the 
general findings from the quantitative portion of the study as well as 
highlighting future areas of research deserving exploration. Representative 
phrases and illuminating remarks from the student and teacher responses were 
included as illustrative examples of the overall findings. 

A panel of five graders was formed, which included a Technology and 
Engineering Education professor, an Engineering Education Professor, an Art 
and Design Professor, a former middle-school teacher, and a graduate student in 
Technology and Engineering Education. The panel of graders was trained on the 
CompareAssess software and discussed the grading procedure together prior to 
completing judgments. Initially, each grader was given a login to the 
CompareAssess online judging platform and was asked to grade 20-30 
portfolios and student products. Following this, an additional meeting was held 
to ensure a unified direction in judgment. Judges were asked to complete 
additional judgments up to 175 judgments of portfolios and 175 judgments of 
products, which resulted in a reliability coefficient of r = .943 for students 
products and r = .934 for student portfolios. Twenty additional judgments for 
portfolios and 20 more judgments for products were completed by each judge, 
which increased the reliability to r = .959 for student products and r = .972 for 
student portfolios. The result was a rank-order for student products and 
portfolios that was added to the statistical data set for later analysis. 

Prior to analyses, regression diagnostics, including linearity, 
homoscedasticity, normality of residuals, uncorrelated error, mean 
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independence, and normally distributed error, were conducted to ensure that the 
proper assumptions of were met for the statistical tests. It was determined that 
each of the tests was satisfied and that the assumptions were met. Following 
this, all quantitative data were analyzed using a variety of statistical procedures, 
including /-test. ANOVA, ANCOVA, correlation, and regression. 

Quantitative Findings 

The quantitative findings from the study were taken from three different 
sources: the pre-study questionnaire (n = 555), the post-study questionnaire (n = 
458), and the matched questionnaire (n = 221) containing student pre- and post¬ 
study matched responses. The large decrease in n-sizc between the pre-study 
questionnaire and the matched questionnaire was due mainly to student error in 
entering identical unique identifiers on both the pre- and the post-study 
questionnaires. In order to ensure comparability between the data, independent 
samples /-tests were computed comparing the pre-study data with the combined 
data set on the following measures to test for significant differences: pre-study 
SDLTS score, DNAS score, average grades, average time spent with 
technology, average mobile device use, and average mobile device skill. The 
only test that revealed a significant difference between the pre-study data set and 
the combined data set was for average grades, F (772) = 6.13 ,p = .023. A 
follow-up independent samples /-test, which compared the grades in TEE 
classes across the groups, did not return significant results (p = .17). These tests 
demonstrate that in all tested cases, with the exception of average grades, the 
students in the combined data set were not significantly different from the total n 
contained in the pre-study data set. It was thus concluded that, although not 
equal, the combined data set is comparable, representative, and suitable for use 
in further data analyses 

Self-Directed Learning Findings 

A simple linear regression was calculated to predict student self-directed 
learning (post-study questionnaire score) based on demographic variables (age, 
grades in all classes, grades in TEE classes, computer and mobile device access, 
time spent with technology, and pre-study SDLTS score). Upon initial 
investigation, it was shown that not all predictors were significant to student 
post-study SDLTS score. Non-significant factors were removed case by case 
until only significant factors were contained in the regression. This resulted in a 
significant regression equation (F (2, 218) = 26.26, p < .001), with an adjusted 
R 2 of. 19, and two significant predictors of student score on the post-study 
SDLTS assessment: average mobile device skill level and computer access and 
use at school (see Table 1). Student post-study SDLTS score is represented by 
2.94 + .40(average mobile device skill level) - ,18(computer access and use at 
school), suggesting a positive correlation between average mobile device skill 
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level in students and self-directed learning and a negative correlation between 
computer access and use at school and student self-directed learning. 

Table 1 


Regression Equation Results for Student Demographic In formation and Post- 
study SDLTS Score _ 


Variable 

Coefficient B 

p-value 

t 

r 

Computer access at 
school 

-.07 

^3 

II 

© 

o 

UJ 

-3.02 

-.18 

Mobile device skill 
level 

.29 

p<. 001 

6.61 

.40 


A paired-samples /-test was used to determine if there was a significant 
difference in the student SDLTS pre- and post-study questionnaires. The results 
evidenced a significant difference in student pre-study (M= 3.61, SD = .54) and 
post-study (M= 3.79, SD = .57) scores, t = 6.521 ,p< .001, d = -.44, indicating 
that students were more self-directed following the study. 

It was anticipated that student scores on the DNAS would be predictive of 
their post-study SDLTS scores. A correlational analysis revealed a significant 
correlation (p < .001) in the positive direction between student DNAS and 
student pre-study SDLTS as well as student post-study SDLTS scores (Table 2), 
suggesting that higher levels of “digital nativeness” among students 
corresponded with higher self-directed learning. 

Table 2 

Correlation for Student DNAS Scores and Student Pre- and Post-Study SDLTS 
Scores 



Pre-SDLTS score 

Post-SDLTS score 

Pearson correlation 

.40 

.31 

Sig. (2-tailed) 

.00 

.00 

n 

221 

221 


Different mediums were purposely utilized for student design portfolios as 
part of the counter-balanced study format. In order to separate significance 
based solely off the difference in portfolio medium, tests were run to determine 
the impact of paper or electronic portfolios on student post-study SDLTS score. 
Utilizing an ANCOVA, with student pre-study SDLTS score as the covariate, 
portfolio type and student post-study SDLTS were analyzed. The resulting p- 
value was not statistically significant {p = .132), suggesting that student post- 
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study SDLTS score was independent of their assigned portfolio creation 
medium. 

Using ANCOVA statistical techniques, analyses were conducted examining 
the relationship between student access to mobile devices and student post-study 
SDLTS score, using students’ pre-study SDLTS score as a covariate. The 
resulting value,/? = .816, was not significant, suggesting that student scores on 
self-directedness in learning with technology are independent of access to 
mobile devices. 

A simple bivariate correlation test was conducted to look at the relationship 
between student comfort level with open-ended design problems and post-study 
SDLTS score. This reflected a significant correlation (p < .001) in the positive 
direction, suggesting that higher comfort levels with open-ended design 
problems are correlated with higher post-study SDLTS scores. 

Using a simple correlation test, the relationship between student comfort 
level in working with groups and student post-study SDLTS scores was found to 
be significant (p < .001) and positive, suggesting that higher comfort in working 
in groups was correlated with higher post-study SDLTS scores. 

Achievement Findings 

Student achievement was measured in two ways as part of this study: 
student rank score on their group portfolio and student rank score on their group 
product (created during the design challenge). Possible relationships between 
student final scores and other potential predictors were explored using a variety 
of statistical methods. 

Using correlation statistical analyses, the relationships between student 
group portfolio score (rank) and student group product score (rank) were 
identified. Table 3 outlines the relationships between student portfolio rank 
score and demographics with several significant (/;<.()5) correlations (age, 
grades, time with technology, skill with mobile devices, and access to mobile 
devices). When compared, the relationship between student product rank score 
and demographics demonstrated that age was the only significant correlation (p 
= .05). 
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Table 3 

Student Demographics Measures and Student Portfolio Rank Score _ 

Student portfolio rank Spearman correlation Sig. (2-tailed) n 


Student age 

Grades in average (all classes) 

Grades on average (TEE only) 
Average time using technology 
Average mobile device use 
Skill level with mobile devices 
DNAS score 
Prestudy SDLTS score 
Computer access (home) and use 
Computer access (school) and use 
Mobile device access (home) and use 
Mobile device access (school) and use 


.16 

.02 

221 

.13 

.05 

220 

-.02 

.83 

221 

.27 

.00 

214 

.05 

.45 

221 

.15 

.02 

221 

.12 

.08 

221 

-.07 

.33 

221 

.05 

.50 

221 

.09 

.17 

218 

.27 

.00 

219 

.24 

.00 

219 


A set of simple correlation tests revealed that the correlation between 
student pre-study SDLTS score and their portfolio rank score was not significant 
(r = -.07, p = .33). The correlation between student pre-study SDLTS score and 
their product rank score was also not significant (r = -.05, p = .48). 

A correlation was computed for student self-directed learning, as measured 
on the post-study SDLTS, and student rank portfolio score. A correlation was 
also computer for student self-directed learning, as measured on the post-study 
SDLTS, and student rank product score. Neither relationship returned a 
significant value. 

Looking at correlation tests, the correlation between student DNAS scores 
and their product rank score was not significant (r = -.04, p = .54). The 
correlation between student DNAS scores and their portfolio rank score 
approached significance (r = .12 ,p = .08) but was not significant. 

Using an independent samples t-test, the impact of portfolio type on student 
achievement (both portfolio and product rank scores) was analyzed. There was a 
significant difference in student product scores between paper (M= 73.93, SD = 
52.22) and electronic portfolios (M= 97.71, SD = 49.63); t (455) = -4.83 ,p< 
.001. There was also a significant difference in student portfolio scores between 
paper (M= 68.83, SD = 39.46) and electronic portfolios (M= 96.58, SD = 
53.43); t (454) = -5.84,/? < .001. It is important to note that the scores for the 
portfolios and the products are rank scores, so a lower rank is deemed of higher 
quality than a higher rank. 

Using an independent samples t-test, the impact of mobile devices on 
student achievement (both portfolio and product rank scores) was analyzed. 
There was a significant difference in student portfolio scores between those with 
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access to mobile devices (M= 81.65, SD = 52.07) and those without access to 
mobile devices (M= 101.29, SD = 42.52); t (454) = -3.62, p< .001. However, 
there was not a significant difference in student product scores between those 
with access to mobile devices (M= 90.20, SD = 52.82) and those without access 
to mobile devices (M= 85.60, SD = 48.60); t (455) = .816,/? = .415. 

A one-way ANOVA was computed to assess the impact of the teacher on 
student achievement scores for the portfolio. The results were significant (F = 
37.70, p < .001), and LSD post hoc analyses were computed to further explore 
the difference between teacher groups (see Table 4). 

Table 4 


Post Hoc Analysis of Differences in Student Product Rank by Teacher 


Teacher (n, M, SD) 1 2 

3 

4 

5 

6 

Teacher 1 (85, 77.99, 47.39) .00 

.00 

.35 

.00 

.79 

Teacher 2 (84, 107.17, 51.11) 

.94 

.00 

.62 

.00 

Teacher 3 (69, 106.54,48.26) 


.00 

.69 

.00 

Teacher 4 (59, 70.10, 47.20) 



.00 

.46 

Teacher 5 (53, 102.85, 44.51) 

Teacher 6 (107, 76.05, 54.89) 




.00 


A separate one-way ANOVA was computed to assess the impact of the 
teacher on student achievement scores for the product, which also returned 
significant results ( F= 8.77, p < .001). This necessitated LSD post hoc analyses 
to further explore the difference between teacher groups (see Table 5). Both 
Tables 4 and 5 highlight significant differences in students achievement based 
on teacher. 

Table 5 


Post Hoc Analysis of Differences in Student Portfolio Rank by Teacher 


Teacher (n,M,SD) 1 

2 3 

4 

5 

6 

Teacher 1 (84, 64.26, 48.98) 

.00 .00 

.00 

.00 

.12 

Teacher 2 (84, 130.55, 44.32) 

.00 

.00 

.00 

.00 

Teacher 3 (69, 85.20, 45.78) 


.22 

.00 

.00 

Teacher 4 (59, 94.58,36.23) 



.08 

.00 

Teacher 5 (53, 108.75, 47.83) 
Teacher 6 (107, 86.47, 50.56) 




.00 
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Qualitative Findings 

The final themes emerging from the student and teacher interviews were 
used to triangulate, clarify, and expand the quantitative findings resulting from 
the analysis of the student questionnaires and final rank order of the products 
and portfolios. The themes, along with representative responses, are included 
here. 

Self-Directed Learning 

Student and teacher comments related to self-directed learning revolved 
around the necessity of student choice for self-directed learning to occur. Two 
students commented on student choice in defining self-directed learning. 

[Self-directed learning is] something that you, like go and do yourself, like 
you are interested in it, you want to go and figure out what this thing is . . . 
or how something works. 

[Self-directed learning is] somebody actually choosing what they have to do 
and what they want to do in their education. 

One teacher comment also highlighted the student initiative and choice 
involved with self-directed learning. 

Self-directed learning is where a student takes their own personal initiative 
to take the supplies that I’ve provided and also the knowledge that 1 have 
provided that they need, and of their own knowledge and their own 
supplies—based off of rules and 1 guess regulations, based off of our 
assignments or whatever—to create a learning environment where they are 
benefitted. 

Mobile Devices 

Teachers and students responded to questions regarding the potential 
benefits and challenges of mobile devices in K-12 education. Their responses 
themed around (a) mobile devices being enablers or both positive and negative 
behavior, (b) mobile devices being regulated by strict rules and monitoring, and 
(c) classroom norms acting in opposition to mobile device integration. Examples 
of student comments related to the enabling nature include the following. 

[Mobile devices] help, because you can look . . . like if you want to learn 
something, like if you were trying to teach yourself how to play the guitar 
or something you could look up videos online of how to do it. 

Well it just matters on the kid pretty much. I think that [mobile devices] 
would help most kids, but some kids are just there to get the grade and to 
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dink off with it and ruin the privileges. It would help them because like, 
they, oh 1 feel familiar with this—I know what to do, I know where to go. 

I think [mobile devices] would help some kids, but some kids would just 
play on them, and then, maybe look up a few things ... 1 would use mine 
for, uh, learning because I don’t really like being on social media, but I 
don’t know about other people very well, I just see a lot of people on 
Instagram during lunch. So . . . I’m not really, out to using it the same way 
as other people. 

Students identified strict rules, regulations, and monitoring as both the 
reality and a necessity for mobile devices in K-12 classrooms. Student 
comments also centered on different areas where mobile devices were allowed 
and other areas where mobile devices were prohibited. Example student 
responses include the following. 

I think [mobile devices] would . . . help, but there would have to be 
restrictions, ‘cuz if kids were just playing on their phones, they wouldn’t be 
learning, and they wouldn’t like, be paying attention to the teacher. So they 
wouldn’t get the grade they want on their test, and, so, that would bring 
grades down, but like using them would in like, effective ways in schools, 
would bring them up. 

In school [mobile devices] are allowed during class, if the teacher gives you 
permission, only if you are, like, working on an assignment or something. 
Um, they are allowed during lunch—private time—before and after school. 
Um, and like usually people just, like, use them to do, like, calculators or 
math, and stuff like that. 

It all depends like what class, like [mobile devices] are not allowed in like, 
during class but some teachers like let you use them for like certain things if 
you don’t know, like, how to like, spell something or like draw something 
then you’re allowed to use them. 

Teachers’ comments were similarly themed to the student responses in 
regards to the need for rules and regulations in order for mobile devices to be 
successful in K-12 classrooms. One example response illustrates this general 
consensus among teachers: 

I think that [mobile devices in K-12 classrooms] can be good in a 
monitored fashion, with activities like the one we did, or other experience 
design activities. It could be very valuable in the research and 
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understanding what the actual problem is they’re trying to solve and where 
it fits in the world of what the impact that decision or solution might have. 

Regardless of permission to use mobile devices, the majority of students did 
not choose to use mobile devices regularly throughout the study. When asked 
about the reasons guiding the students’ decision to use or not to use a mobile 
device during the study, teachers highlighted classroom norms as a potential 
reason for students not using mobile device: Students and teachers were 
accustomed to a restriction on device use in class that prevailed despite 
permission to use the devices. One teacher remarked, 

I had a couple kids looking on the iPad on the Internet. Honestly I was 
surprised that when we opened it up to the mobile devices more students 
didn’t have their cell phones out. Most of them were just looking for images 
in [one] of the pill bottle folder things. But I was surprised at, I guess, the 
lack of using that device. Maybe it’s because they’re not used to using it in 
my classroom—I really don’t know. The only thing I can think of is 
because it’s the rule that you don’t have your cell phone out in my class, I 
kind of felt like that was the norm. 

Summary of Findings 

For the middle-school students in this study, self-directed learning appeared 
to be related to student and environmental characteristics rather than access to 
specific technology tools. When analyzed, student self-directed learning was 
independent and even negatively correlated with access to some technology 
tools (e.g., mobile devices and computers), and student self-directedness in 
learning scores were independent of student portfolio type (paper vs. electronic). 
Taken together, these findings suggest that technology tools in and of 
themselves may not correspond with an increase in student self-directed learning 
and, in some cases, may be detrimental to student self-directed learning. These 
findings appear to align with Mentzer’s (2011) research, which also concluded 
that access to information (i.e., the Internet via computers) did not significantly 
improve student designs. 

Unlike technology tools, a variety of specific student and classroom 
environment characteristics did demonstrate significant relationships with 
student self-directed learning. Student characteristics that corresponded with 
higher levels of self-directedness in learners were: average skill in using mobile 
devices, higher “digital nativeness” scores, student familiarity with open-ended 
design problems, and student comfort level in working in groups. Responses in 
teacher interviews seem to concur with this. Teachers discussed how they 
perceived self-directed learning to be a product of external conditions such as: 
the presence of an open-ended problem, a task involving group work, or other 
classroom-environmental factors. 
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Student achievement was identified through two separate student scores: 
student portfolio scores and student product scores. In qualitative interviews, the 
teachers and students were in agreement that mobile devices had the potential to 
improve students’ achievement if used correctly. 

A key finding is that teachers and portfolio medium (paper vs. electronic) 
were the most significant factors in student achievement. Students completing 
portfolios on paper produced significantly better portfolios and products than 
their counterparts who completed electronic portfolios. Despite the fact that all 
teachers in the study were comparable, there were significant differences in the 
final grades received by the students of each teacher, with the students of one 
teacher (Teacher 6) scoring significantly higher than the other students in the 
study. This teacher was assigned to complete the portfolios on paper, which may 
be a confounding factor resulting in the paper portfolios being ranked much 
higher than the electronic portfolios and this teacher’s students outperforming 
the others. These findings align with other research demonstrating the 
significant impact of a teacher on their students above and beyond other factors 
(Darling-Hammond, 2000). 

Student Portfolios 

Student access to mobile devices was significantly correlated with higher 
study scores on the design portfolio. Average time spent with technology, 
student age, mobile device skill level, and mobile device access at home and 
school were also significantly correlated with higher scores. Student pre-study 
SDLTS and student post-study SDLTS scores were both independent of student 
portfolio score rank—an important finding suggesting that self-directed learning 
may not be indicative of student achievement, ability, and skill with the 
engineering design process despite its identification as a key skill for 21st 
century learners (Partnership for 21st Century Learning, 2015). 

Student Products 

Unlike the portfolio scores, the only significant correlation found between 
student product scores, aside from teacher and portfolio type, was student age. 
Older students tended to receive better scores on their design products. Student 
portfolio scores were not significantly correlated with pre- or post-study 
SDLTS, pre-study DNAS score, or access to mobile devices. Interestingly, the 
two teachers with the overall top-performing students (Teachers 4 and 6) had the 
youngest average students in their participating classes across the study. This 
emphasizes the strength of the impact made on students’ achievement by their 
teacher. 

Other Observations 

Of particular interest, the researcher noticed that although many students 
were given access to mobile devices, students rarely used mobile devices during 
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the product creation or the portfolio creation. Teachers echoed this sentiment 
during interviews and provided several conjectures for lack of mobile device 
usage, including: lack of need for mobile devices, the competition between 
computers and mobile devices, and classroom norms. Although students cited 
specific benefits of mobile devices in the interviews, the majority (65.4%) of 
students who were given access to mobile devices during the study reported 
using mobile devices less than 30 minutes during class over the course of the 
entire study (over 360 minutes of class time). In the interviews, students 
mentioned that this activity was the “wrong type of problem” for using a mobile 
device. When asked for clarification, students commented that they were unsure 
how to use a mobile device to help them with an open-ended problem and were 
most comfortable using their mobile devices to answer factual single-answer 
problems. Student and teacher interview responses also themed around 
classroom norms: Although mobile devices were allowed, the previously 
established classroom norm (no mobile devices allowed) appeared to be highly 
influential on students’ choices regarding mobile device use. 

Further Research and Analysis 

Using both the quantitative and qualitative findings from this study, the 
following recommendations and areas for further research and analysis were 
identified. 

Self-Directed Learning 

As noted, mobile devices did not make a significant impact on student self- 
directed learning as measured by the SDLTS on the pre- or post-study 
questionnaires. However, several other student and classroom environment 
characteristics were positively correlated with self-directed learning in a 
significant way, including student skill in using mobile devices and student 
“digital nativeness.” This suggests that teachers and schools should emphasize 
student skills in using and interacting with technology as a means of improving 
self-directed learning. If students can more effectively interact with different 
technologies (e.g., mobile devices, tablets, and computers), their opportunities 
and abilities for self-directed learning may also increase. 

Mobile Devices 

Although mobile devices did not significantly impact student self-directed 
learning in this study, mobile devices did correlate significantly with higher 
student achievement on the design portfolio. During student interviews, a theme 
that emerged with relation to mobile devices was the need for direct instruction 
regarding how, when, and where students should use mobile devices. Teachers 
can work through explicit instruction so those students understand how to use 
mobile devices and so that positive and appropriate uses of mobile devices 
become the new norm for their classroom. 
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Further Research 

Additional relationships between student mobile device access and factors 
outside of self-directed learning or achievement would shed further light on the 
debate over mobile devices in the classroom. The findings from this study are 
limited in scope to a relatively suburban, middle-class, homogeneous population 
within specific grade levels (seventh and eighth grade). Other research with 
different population groups, ages, or different locations could shed additional 
light on and provide valuable comparisons for the findings of this study. As 
teacher impact was highly significant in this study, it is recommended that 
additional studies of varying research designs be undertaken to explore 
supplementary data that examine specifically the impacts of teacher influence. 
Additionally, student gender was not collected during this study—this has been 
identified in other studies as significant (Reio & Davis, 2005) and should be 
taken into account in future research efforts around student self-directed 
learning. 


Implications 

Granting access to mobile devices in middle school TEE classrooms during 
a STEM activity appears to have the potential to transform and improve student 
educational experiences. Although student self-directed learning was not 
significantly impacted in this study by access to mobile devices, aspects of 
student achievement showed positive correlations with access to mobile devices. 
In order for mobile devices to be impactful, teachers and students will need to 
work together to change the classroom norms relating to mobile device use, and 
teachers will need to model appropriate and effective mobile device use for their 
students. 

In this study, student self-directed learning correlated more closely with 
student and classroom characteristics than it did with access to technology tools, 
suggesting a possible shift in the debate surrounding mobile device inclusion in 
classrooms from the actual tools to the learner and classroom characteristics. As 
previously shown (Darling-Hammond, 2000), the impact of a teacher on student 
achievement is significant: Students’ final portfolio and product scores were 
more directly related to their teacher than any other variable. Focus on effective 
teacher habit identification and training should take precedence over technology 
tools and other classroom add-ons. 
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